The Boolean Language of Thought is recoverable from learning data.

According to the Language of Thought Hypothesis (LoTH), an influential account in philosophy and cognitive science, human cognition is underlain by symbolic reasoning in a formal language. In this account, concepts are expressions in a Language of Thought, deduction is syntactic manipulation in this language, and learning is an inference of expressions in this language from data. This picture raises the question of what LoT humans have, and how to infer it from behavior. In this paper, we pave the way towards answering this question, by approaching a more fundamental question: to what extent is it possible in principle to recover the human LoT from experimental data? To answer this question, we focus on the fragment of LoT that is concerned with representing Boolean categories and simulate the recovery of the Boolean LoT from category learning experiments. Our findings show that in principle the vast majority of Boolean LoTs can be accurately recovered from experimental data. However, we find that this crucially depends on the employed experimental design. Moreover, we find evidence that LoTs with fewer operators can be recovered from category learning data faster.

A systematic investigation into the reliability of inter-temporal choice model parameters.

Decades of work have been dedicated to developing and testing models that characterize how people make inter-temporal choices. Although parameter estimates from these models are often interpreted as indices of latent components of the choice process, little work has been done to examine their reliability. This is problematic because estimation error can bias conclusions that are drawn from these parameter estimates. We examine the reliability of parameter estimates from 11 prominent models of inter-temporal choice by (a) fitting each model to data from three previous experiments with designs representative of those typically used to study inter-temporal choice, (b) examining the consistency of parameters estimated for the same person based on different choice sets, and (c) conducting a parameter recovery analysis. We find generally low correlations between parameters estimated for the same person from the different choice sets. Moreover, parameter recovery varies considerably between models and the experimental designs upon which parameter estimates are based. We conclude that many parameter estimates reported in previous research are likely unreliable and provide recommendations on how to enhance the reliability of inter-temporal choice models for measurement purposes.

A cognitive process modeling framework for the ABCD study stop-signal task.

The Adolescent Brain Cognitive Development (ABCD) Study is a longitudinal neuroimaging study of unprecedented scale that is in the process of following over 11,000 youth from middle childhood though age 20. However, a design feature of the study's stop-signal task violates "context independence", an assumption critical to current non-parametric methods for estimating stop-signal reaction time (SSRT), a key measure of inhibitory ability in the study. This has led some experts to call for the task to be changed and for previously collected data to be used with caution. We present a cognitive process modeling framework, the RDEX-ABCD model, that provides a parsimonious explanation for the impact of this design feature on "go" stimulus processing and successfully accounts for key behavioral trends in the ABCD data. Simulation studies using this model suggest that failing to account for the context independence violations in the ABCD design can lead to erroneous inferences in several realistic scenarios. However, we demonstrate that RDEX-ABCD effectively addresses these violations and can be used to accurately measure SSRT along with an array of additional mechanistic parameters of interest (e.g., attention to the stop signal, cognitive efficiency), advancing investigators' ability to draw valid and nuanced inferences from ABCD data. AVAILABILITY OF DATA AND MATERIALS: Data from the ABCD Study are available through the NIH Data Archive (NDA): nda.nih.gov/abcd. Code for all analyses featured in this study is openly available on the Open Science Framework (OSF): osf.io/2h8a7/.

Examining the Robustness of the Graded Response and 2-Parameter Logistic Models to Violations of Construct Normality.

When fitting unidimensional item response theory (IRT) models, the population distribution of the latent trait (θ) is often assumed to be normally distributed. However, some psychological theories would suggest a nonnormal θ. For example, some clinical traits (e.g., alcoholism, depression) are believed to follow a positively skewed distribution where the construct is low for most people, medium for some, and high for few. Failure to account for nonnormality may compromise the validity of inferences and conclusions. Although corrections have been developed to account for nonnormality, these methods can be computationally intensive and have not yet been widely adopted. Previous research has recommended implementing nonnormality corrections when θ is not "approximately normal." This research focused on examining how far θ can deviate from normal before the normality assumption becomes untenable. Specifically, our goal was to identify the type(s) and degree(s) of nonnormality that result in unacceptable parameter recovery for the graded response model (GRM) and 2-parameter logistic model (2PLM).

A comparative evaluation of factor- and component-based structural equation modelling approaches under (in)correct construct representations.

Structural equation modelling (SEM) has evolved into two domains, factor-based and component-based, dependent on whether constructs are statistically represented as common factors or components. The two SEM domains are conceptually distinct, each assuming their own population models with either of the statistical construct proxies, and statistical SEM approaches should be used for estimating models whose construct representations correspond to what they assume. However, SEM approaches have often been evaluated and compared only under population factor models, providing misleading conclusions about their relative performance. This is partly because population component models and their relationships have not been clearly formulated. Also, it is of fundamental importance to examine how robust SEM approaches can be to potential misrepresentation of constructs because researchers may often lack clear theories to determine whether a factor or component is more representative of a given construct. Addressing these issues, this study begins by clarifying several population component models and their relationships and then provides a comprehensive evaluation of four SEM approaches - the maximum likelihood approach and factor score regression for factor-based SEM as well as generalized structured component analysis (GSCA) and partial least squares path modelling (PLSPM) for component-based SEM - under various experimental conditions. We confirm that the factor-based SEM approaches should be preferred for estimating factor models, whereas the component-based SEM approaches should be chosen for component models. Importantly, the component-based approaches are generally more robust to construct misrepresentation than the factor-based ones. Of the component-based approaches, GSCA should be chosen over PLSPM, regardless of whether or not constructs are misrepresented.

Comparing Eight Parameter Estimation Methods for the Ratcliff Diffusion Model Using Free Software.

The Ratcliff Diffusion Model has become an important and widely used tool for the evaluation of psychological experiments. Concurrently, numerous programs and routines have appeared to estimate the model's parameters. The present study aims at comparing some of the most widely used tools with special focus on freely available routines (i.e., open source). Our simulations show that (1) starting point and non-decision time were recovered better than drift rate, (2) the Bayesian approach outperformed all other approaches when the number of trials was low, (3) the Kolmogorov-Smirnov and χ2 approaches revealed more bias than Bayesian or Maximum Likelihood based routines, and (4) EZ produced substantially biased estimates of threshold separation, non-decision time and drift rate when starting point z ≠ a/2. We discuss the implications for the choice of parameter estimation approaches for real data and suggest that if biased starting point cannot be excluded, EZ will produce deviant estimates and should be used with great care.

Improving parameter recovery for conflict drift-diffusion models.

Several drift-diffusion models have been developed to account for the performance in conflict tasks. Although a common characteristic of these models is that the drift rate changes within a trial, their architecture is rather different. Comparative studies usually examine which model fits the data best. However, a good fit does not guarantee good parameter recovery, which is a necessary condition for a valid interpretation of any fit. A recent simulation study revealed that recovery performance varies largely between models and individual parameters. Moreover, recovery was generally not very impressive. Therefore, the aim of the present study was to introduce and test an improved fit procedure. It is based on a grid search for determining the initial parameter values and on a specific criterion for assessing the goodness of fit. Simulations show that not only the fit performance but also parameter recovery improved substantially by applying this procedure, compared to the standard one. The improvement was largest for the most complex model.

A parameter recovery assessment of time-variant models of decision-making.

Evidence accumulation models have been one of the most dominant modeling frameworks used to study rapid decision-making over the past several decades. These models propose that evidence accumulates from the environment until the evidence for one alternative reaches some threshold, typically associated with caution, triggering a response. However, researchers have recently begun to reconsider the fundamental assumptions of how caution varies with time. In the past, it was typically assumed that levels of caution are independent of time. Recent investigations have however suggested the possibility that levels of caution decrease over time and that this strategy provides more efficient performance under certain conditions. Our study provides the first comprehensive assessment of this newer class of models accounting for time-varying caution to determine how robustly their parameters can be estimated. We assess five overall variants of collapsing threshold/urgency signal models based on the diffusion decision model, linear ballistic accumulator model, and urgency gating model frameworks. We find that estimation of parameters, particularly those associated with caution/urgency modulation are most robust for the linearly collapsing threshold diffusion model followed by an urgency-gating model with a leakage process. All other models considered, particularly those with ballistic accumulation or nonlinear thresholds, are unable to recover their own parameters adequately, making their usage in parameter estimation contexts questionable.

GGUM-RANK Statement and Person Parameter Estimation With Multidimensional Forced Choice Triplets.

Historically, multidimensional forced choice (MFC) measures have been criticized because conventional scoring methods can lead to ipsativity problems that render scores unsuitable for interindividual comparisons. However, with the recent advent of item response theory (IRT) scoring methods that yield normative information, MFC measures are surging in popularity and becoming important components in high-stake evaluation settings. This article aims to add to burgeoning methodological advances in MFC measurement by focusing on statement and person parameter recovery for the GGUM-RANK (generalized graded unfolding-RANK) IRT model. Markov chain Monte Carlo (MCMC) algorithm was developed for estimating GGUM-RANK statement and person parameters directly from MFC rank responses. In simulation studies, it was examined that how the psychometric properties of statements composing MFC items, test length, and sample size influenced statement and person parameter estimation; and it was explored for the benefits of measurement using MFC triplets relative to pairs. To demonstrate this methodology, an empirical validity study was then conducted using an MFC triplet personality measure. The results and implications of these studies for future research and practice are discussed.

Improved information pooling for hierarchical cognitive models through multiple and covaried regression.

Cognitive process models are fit to observed data to infer how experimental manipulations modify the assumed underlying cognitive process. They are alternatives to descriptive models, which only capture differences on the observed data level, and do not make assumptions about the underlying cognitive process. Process models may require more observations than descriptive models however, and as a consequence, usually fewer conditions can be simultaneously modeled with them. Unfortunately, it is known that the predictive validity of a model may be compromised when fewer experimental conditions are jointly accounted for (e.g., overestimation of predictor effects, or their incorrect assignment). We develop a hierarchical and covaried multiple regression approach to address this problem. Specifically, we show how to map the recurrences of all conditions, participants, items, and/or traits across experimental design cells to the process model parameters. This systematic pooling of information can facilitate parameter estimation. The proposed approach is particularly relevant for multi-factor experimental designs, and for mixture models that parameterize per cell to assess predictor effects. This hierarchical framework provides the capacity to model more conditions jointly to improve parameter recovery at low observation numbers (e.g., using only 1/6 of trials, recovering as well as standard hierarchical Bayesian methods), and to directly model predictor and covariate effects on the process parameters, without the need for post hoc analyses (e.g., ANOVA). An example application to real data is also provided.

Parameter recovery, bias and standard errors in the linear ballistic accumulator model.

The linear ballistic accumulator (LBA) model (Brown & Heathcote, , Cogn. Psychol., 57, 153) is increasingly popular in modelling response times from experimental data. An R package, glba, has been developed to fit the LBA model using maximum likelihood estimation which is validated by means of a parameter recovery study. At sufficient sample sizes parameter recovery is good, whereas at smaller sample sizes there can be large bias in parameters. In a second simulation study, two methods for computing parameter standard errors are compared. The Hessian-based method is found to be adequate and is (much) faster than the alternative bootstrap method. The use of parameter standard errors in model selection and inference is illustrated in an example using data from an implicit learning experiment (Visser et al., , Mem. Cogn., 35, 1502). It is shown that typical implicit learning effects are captured by different parameters of the LBA model.

Bayesian Modal Estimation of the Four-Parameter Item Response Model in Real, Realistic, and Idealized Data Sets.

In this study, we explored item and person parameter recovery of the four-parameter model (4PM) in over 24,000 real, realistic, and idealized data sets. In the first analyses, we fit the 4PM and three alternative models to data from three Minnesota Multiphasic Personality Inventory-Adolescent form factor scales using Bayesian modal estimation (BME). Our results indicated that the 4PM fits these scales better than simpler item Response Theory (IRT) models. Next, using the parameter estimates from these real data analyses, we estimated 4PM item parameters in 6,000 realistic data sets to establish minimum sample size requirements for accurate item and person parameter recovery. Using a factorial design that crossed discrete levels of item parameters, sample size, and test length, we also fit the 4PM to an additional 18,000 idealized data sets to extend our parameter recovery findings. Our combined results demonstrated that 4PM item parameters and parameter functions (e.g., item response functions) can be accurately estimated using BME in moderate to large samples (N ⩾ 5, 000) and person parameters can be accurately estimated in smaller samples (N ⩾ 1, 000). In the supplemental files, we report annotated [Formula: see text] code that shows how to estimate 4PM item and person parameters in [Formula: see text] (Chalmers, 2012 ).

Comparative Analyses of MIRT Models and Software (BMIRT and flexMIRT).

Application of MIRT modeling procedures is dependent on the quality of parameter estimates provided by the estimation software and techniques used. This study investigated model parameter recovery of two popular MIRT packages, BMIRT and flexMIRT, under some common measurement conditions. These packages were specifically selected to investigate the model parameter recovery of three item parameter estimation techniques, namely, Bock-Aitkin EM (BA-EM), Markov chain Monte Carlo (MCMC), and Metropolis-Hastings Robbins-Monro (MH-RM) algorithms. The results demonstrated that all estimation techniques had similar root mean square error values when larger sample size and higher test length were used. Depending on the number of dimensions, sample size, and test length, each estimation technique exhibited some strengths and weaknesses. Overall, the BA-EM technique was found to have shorter estimation time with all test specifications.

Sample Size Requirements for Estimation of Item Parameters in the Multidimensional Graded Response Model.

Likert types of rating scales in which a respondent chooses a response from an ordered set of response options are used to measure a wide variety of psychological, educational, and medical outcome variables. The most appropriate item response theory model for analyzing and scoring these instruments when they provide scores on multiple scales is the multidimensional graded response model (MGRM) A simulation study was conducted to investigate the variables that might affect item parameter recovery for the MGRM. Data were generated based on different sample sizes, test lengths, and scale intercorrelations. Parameter estimates were obtained through the flexMIRT software. The quality of parameter recovery was assessed by the correlation between true and estimated parameters as well as bias and root-mean-square-error. Results indicated that for the vast majority of cases studied a sample size of N = 500 provided accurate parameter estimates, except for tests with 240 items when 1000 examinees were necessary to obtain accurate parameter estimates. Increasing sample size beyond N = 1000 did not increase the accuracy of MGRM parameter estimates.

Precisión en la recuperación de parámetros, con datos ordinales, en el Análisis de Estructura de Covarianza y el Modelo de Rutas mediante Mínimos Cuadrados Parciales / Accuracy on Parameter Recovery, with Ordinals Data, of Structure Covariance Analysis and Partial Least Squares Path Modeling

Se compara la precisión en la recuperación de parámetros del Análisis de Estructura de Covarianza (ACOV) y el Modelo de Rutas mediante Mínimos Cuadrados Parciales (PLS-PM), en un modelo simple con variables manifiestas simuladas con escala ordinal de cinco puntos. Se utiliza un diseño experimental, manipulando el método de estimación, tamaño muestral, nivel de asimetría y tipo de especificación del modelo. Se valora la media de las diferencias absolutas para el modelo estructural. ACOV presenta estimaciones más precisas que PLS-PM, en distintas condiciones experimentales. Cuando se utiliza un tamaño muestral pequeño, ambas técnicas son igualmente precisas. Se sugiere utilizar ACOV frente a PLS-PM. Se desaconseja fundamentar la elección de PLS-PM frente a ACOV en la utilización de una muestra pequeña.

The accuracy on parameter recovery is compared between Structure Covariance Analysis (ACOV) and Partial Least Squares Path Modeling (PLS-PM), with simulated ordinals data with 5 points, in a simple model. An experimental design is used, controlling the estimation method, sample size, skewness level and model specification. Mean absolute differences are used to assess accuracy for the structural model. ACOV provided more accurate estimates of the structural parameters than PLS-PM in different experimental conditions. With a small sample size, both techniques are equally accurate. Using ACOV against PLS -PM is suggested. PLS choosing ACOV instead based on the use of a small sample size is not recommended.

Parameter recovery for the time-window-of-integration (TWIN) model of multisensory integration in focused attention.

In multisensory settings such as the focused attention paradigm (FAP), subjects are instructed to respond to stimuli of the target modality only, yet reaction times tend to be shorter if an unattended stimulus is presented within a certain spatiotemporal vicinity of the target. The time window of integration (TWIN) model predicts successfully these observed cross-modal reaction time effects. It proposes that all the initially unimodal information must arrive at a point of integration within a certain time window in order to be integrated and thus to initiate response enhancements like the observed reaction time reductions. Here we conducted a parameter recovery study of the TWIN model for focused attention tasks, with five parameters (the durations of the visual and auditory unimodal and the integrated second stage, the width of the time window, and the effect size). Results show that parameter estimates were highly accurate (unbiased, constant error less than 5 ms) and precise (variable error less than 8 ms) throughout, speaking to a high reliability and criterion validity of the process. Further analyses ensured that the estimation procedure is consistent and sufficiently robust against contamination (faulty integration). It can thus be used to estimate reliably the point of integration and the width of the time window.